Screenshot capture handler

Captures full-page screenshots for visual context in Gemini prompts. Stitching long pages, data URL handling, and failure modes are the main topics.

The screenshot capture system follows a layered architecture pattern :

graph TB
subgraph "UI Layer"
SP[Side Panel UI]
SC[Solve Controller]
end
subgraph "Message Layer"
RT[Runtime Adapter]
MR[Message Router]
MT[Message Types]
end
subgraph "Background Service"
SH[Screenshot Handler]
SS[Screenshot Service]
TS[Tabs Adapter]
SA[Scripting Adapter]
end
subgraph "Platform Layer"
BT[Browser Tabs API]
BS[Browser Scripting API]
end
SP --> SC
SC --> RT
RT --> MR
MR --> SH
SH --> SS
SS --> TS
SS --> SA
TS --> BT
SA --> BS
MT -.-> SH
MT -.-> SS

Components stay loosely coupled with a straightforward data flow. The system handles cross-browser compatibility through platform adapters and provides reliable error handling mechanisms.

The screenshot generation process uses a detailed multi-step approach built to handle long web pages efficiently:

sequenceDiagram
participant UI as UI Controller
participant BG as Background Handler
participant SS as Screenshot Service
participant TS as Tabs Adapter
participant SA as Scripting Adapter
participant BA as Browser API
UI->>BG : CAPTURE_FULL_PAGE(tabId, windowId)
BG->>SS : captureFullPage(tabId, windowId)
SS->>SA : executeScript(getDimensions)
SA->>BA : Get page dimensions
BA-->>SA : Dimensions data
SA-->>SS : scrollInfo
loop For each viewport
SS->>SA : executeScript(scrollTo(y))
SA->>BA : Scroll to position
SS->>TS : captureVisibleTab(jpeg, quality : 60)
TS->>BA : Capture screenshot
BA-->>TS : Data URL
TS-->>SS : Base64 image data
SS->>SS : Extract base64 from data URL
end
SS->>SA : executeScript(restoreScroll)
SA->>BA : Restore original position
SS-->>BG : Array of screenshot objects
BG-->>UI : {screenshots : [...]}

The process begins by analyzing the page dimensions to determine the number of viewport captures needed. The system calculates the optimal number of screenshots based on the page height and viewport dimensions, with intelligent limiting to prevent excessive API usage.

The screenshot capture system implements a standardized encoding and response format that keeps compatibility across different browsers and platforms:

Data URL processing#

The system captures screenshots in JPEG format and processes them through a structured data URL parsing mechanism:

Response structure#

The handler returns a detailed response object containing all captured screenshots with metadata:

classDiagram
class ScreenshotResponse {
+Screenshot[] screenshots
}
class Screenshot {
+string mimeType
+string base64
+number scrollY
+number index
+number total
}
ScreenshotResponse --> Screenshot : contains

The response shape matches what Gemini expects: image bytes plus enough metadata to keep context.

Screenshot capture currently uses one image format with fixed quality settings:

Image format specifications#

Format selection rationale#

The JPEG format was chosen for several strategic reasons:

1. Compression Efficiency: Reduces file size significantly and keeps acceptable quality
2. Universal Support: Ensures compatibility across all supported browsers
3. Memory Efficiency: Lower memory footprint compared to PNG or WebP formats
4. AI Optimization: Balanced quality-to-size ratio suitable for AI processing

Quality impact analysis#

The selected quality setting of 60 provides an optimal balance between:

• File Size: Reduced bandwidth and storage requirements
• Visual Clarity: Sufficient detail for AI recognition tasks
• Processing Speed: Faster transmission and processing times

Size limits stack so captures stay within browser and API caps:

Per-Screenshot limits#

Intelligent limiting mechanism#

The system calculates the required number of screenshots and applies intelligent limiting:

flowchart TD
Start([Calculate Screenshots]) --> GetDimensions["Get Page Dimensions"]
GetDimensions --> CalcNum["Calculate Required Screenshots"]
CalcNum --> NumScreenshots["numScreenshots = ceil(scrollHeight/clientHeight)"]
NumScreenshots --> CheckLimit{"numScreenshots > 8?"}
CheckLimit --> |Yes| ApplyLimit["maxScreenshots = 8"]
CheckLimit --> |No| UseActual["maxScreenshots = numScreenshots"]
ApplyLimit --> Proceed["Proceed with Limited Capture"]
UseActual --> Proceed
Proceed --> End([Capture Complete])

Memory management#

The system implements careful memory management to prevent browser crashes:

• Sequential Processing: Screenshots are captured one at a time
• Automatic Cleanup: Previous scroll positions are restored
• Graceful Degradation: Partial failures don’t halt the entire process

The screenshot capture handler integrates smoothly with the side panel UI through a well-defined communication protocol:

UI preview integration#

sequenceDiagram
participant SP as Side Panel
participant SC as Solve Controller
participant BG as Background Handler
participant UI as UI Elements
SP->>SC : User clicks "Solve Assignment"
SC->>BG : Request screenshots
BG-->>SC : {screenshots : [...]}
SC->>UI : Update preview area
UI->>UI : Render thumbnail grid
UI->>UI : Enable screenshot navigation
Note over SP,UI : Real-time preview updates

Communication protocol#

The integration uses the established message-based communication system:

UI state management#

The side panel maintains synchronized state during screenshot operations:

• Loading States: Visual feedback during capture
• Error Handling: Graceful degradation on failures
• Preview Updates: Real-time thumbnail rendering

The screenshot capture system implements detailed error handling strategies so operation stays reliable:

Error categories and handling#

Reliability features#

The system incorporates several built-in safeguards:

1. Graceful Degradation: Screenshots are optional for AI processing
2. Partial Success: Individual screenshot failures don’t halt the entire process
3. Resource Management: Automatic cleanup of temporary resources
4. Logging: Detailed error tracking for debugging

Error recovery mechanisms#

flowchart TD
Error[Error Occurs] --> CheckType{Error Type}
CheckType --> |Permission| WarnUser["Show permission warning"]
CheckType --> |API Limit| ReduceQuality["Reduce quality/limit count"]
CheckType --> |Memory| Cleanup["Cleanup resources"]
CheckType --> |Other| LogError["Log and report"]
WarnUser --> Continue["Continue without screenshots"]
ReduceQuality --> Retry["Retry with adjusted settings"]
Cleanup --> Retry
LogError --> Report["Report to user"]
Continue --> Success([Success])
Retry --> Success
Report --> Failure([Failure])

Basic screenshot capture#

The most common usage scenario involves capturing full-page screenshots for assignment analysis:

// Example: Capturing screenshots for an NPTEL assignment
const screenshotResult = await sendMessageWithRetry(
 runtime,
 {
 type: MESSAGE_TYPES.CAPTURE_FULL_PAGE,
 tabId: targetTabId,
 windowId: targetWindowId,
 },
 { maxRetries: 3, baseDelay: 200 }
);

// Process the returned screenshots
const screenshots = screenshotResult?.screenshots || [];
console.log(`Captured ${screenshots.length} screenshots`);

Integration with AI processing#

Screenshots are smoothly integrated with the Gemini AI service:

// Example: Using screenshots with AI extraction
const extraction = await gemini.extract(
 apiKey,
 pageData.html,
 pageInfo,
 pageData.images || [],
 screenshots, // ← Screenshots passed here
 model,
 reasoningLevel
);

Error handling implementation#

// Example: screenshot capture with error handling
let screenshots = [];
try {
 const screenshotResult = await this.captureFullPageScreenshots(
 targetTabId,
 pageData.windowId
 );
 screenshots = screenshotResult?.screenshots || [];
} catch (ssError) {
 logger.log(`Screenshot capture failed: ${ssError.message}`);
 // Continue without screenshots
}

Performance notes for common capture cases:

Memory optimization#

• Streaming Capture: Screenshots are processed individually to minimize memory usage
• Automatic Cleanup: Previous scroll positions are restored immediately
• Resource Limits: Built-in caps prevent excessive resource consumption

Network efficiency#

• JPEG Compression: Optimized compression reduces bandwidth usage
• Quality Tuning: Balanced quality vs. size ratio
• Batch Processing: Intelligent batching prevents API throttling

Browser compatibility#

The system adapts to different browser capabilities:

• Chrome: Full feature support with advanced APIs
• Firefox: Graceful degradation with alternative approaches
• Cross-Browser: Consistent behavior across platforms

Common issues and solutions#

Debugging steps#

1. Enable Debug Logging: Check browser console for detailed error messages
2. Test Permissions: Verify extension has screenshot permissions
3. Validate Page State: Ensure target page is fully loaded
4. Monitor Resources: Watch for memory usage spikes
5. Check Network: Verify internet connectivity for AI processing

Performance monitoring#

The system provides detailed logging for troubleshooting:

• Operation Timing: Capture duration and processing times
• Error Tracking: Detailed error messages and stack traces
• Resource usage. Memory and CPU use
• Success Rates: Capture success statistics over time

Long pages are scrolled and stitched. Prefer screenshots when text extraction alone loses diagrams or equations.